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Abstract 


Defense  acquisition  is  characterized  by  significant  levels  of  risk  throughout  the 
lifecycle.  Risk,  of  course,  may  result  in  undesirable  outcomes.  Deriving  from  many 
sources,  both  technical  and  organizational,  risk  is  inherently  a  socio-technical 
phenomenon  in  enterprises  such  as  acquisition.  As  such,  it  is  difficult  to  address.  At 
the  same  time,  fiscal  pressures  are  causing  decreased  funding  and  increased 
expectations  for  acquisition  performance.  This  points  to  the  importance  of  risk 
characterization  and  mitigation.  Our  previous  work  has  focused  on  using  simulation 
to  model  and  analyze  acquisition  processes  and  incentives,  to  understand  how  they 
can  be  designed  to  improve  outcomes.  Traditional  simulation  analysis  is  not  well- 
suited  to  modeling  the  socio-technical  complexities  of  risk  in  a  systematic  way, 
though.  This  paper  presents  a  decision/event  network  construct  implemented  within 
enterprise  simulation  models  to  represent  the  complexities  of  risk  over  time.  The  F- 
35  Joint  Strike  Fighter  program  is  analyzed  with  respect  to  risk  and  potential 
outcomes  using  this  enterprise  simulation  framework.  The  F-35  program  embodies 
a  number  of  transformative  approaches  to  the  defense  acquisition  enterprise.  Such 
enterprise  transformation  efforts  are  prone  to  substantial  risk.  In  particular,  cost  and 
schedule  risks  and  outcomes  are  studied  relative  to  program  design  decisions.  Risk 
mitigation  strategies  are  identified  and  presented. 

Keywords:  Enterprise  simulation,  risk,  enterprise  transformation,  F-35  Joint  Strike 
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Addressing  Risk  in  the  Acquisition  Lifecycle 
With  Enterprise  Simulation 


Introduction 

Defense  acquisition  is  characterized  by  significant  levels  of  risk  throughout 
the  lifecycle  of  new  system  development,  production,  and  sustainment. 
Fundamentally,  risk  results  from  the  combination  of  an  uncertain  future 
(probabilities)  and  its  potential  bad  outcomes  (magnitudes).  Classic  risk  drivers  in 
acquisition  include  immature  technologies,  overly  optimistic  baseline  cost  and 
schedule  estimates,  overly  stringent  requirements,  poorly  understood  implicit 
requirements,  and  changing  missions  and  environments.  Increasingly,  risks  come 
from  new  sources,  such  as  fiscal  pressures  on  government  spending  and 
transformative  initiatives  in  the  acquisition  enterprise. 

This  paper  presents  an  enterprise  simulation  approach  to  analyzing  risk  in 
major  acquisition  programs.  This  approach  models  interactions  between  firms  and 
agencies  within  the  acquisition  enterprise,  and  it  also  includes  a  decision/event 
network  that  is  issued  to  characterize  the  relationships  of  risk  drivers  to  outcomes. 
The  approach  is  used  to  model  the  acquisition  of  the  F-35  Joint  Strike  Fighter,  a 
large-scale  acquisition  program  that  has  seen  effects  from  risk.  The  F-35  program 
embodies  a  number  of  transformative  approaches  to  the  defense  acquisition 
enterprise.  Such  enterprise  transformation  efforts  are  prone  to  substantial  risk.  The 
focus  is  on  the  decision/event  network  and  its  use  in  characterizing  risks.  This 
program  is  currently  in  a  combination  of  system  design  and  development  and  low- 
rate  initial  production. 

The  remainder  of  this  paper  is  organized  as  follows.  The  next  section, 
Simulation  of  Acquisition  Programs  and  Risk,  describes  the  use  of  simulation  in 
modeling  and  analyzing  acquisition  programs,  with  a  focus  on  the  uncertainties 
involved  in  such  programs.  Enterprise  Simulation  and  Acquisition  Risk  Analysis 
introduces  the  methodology  of  enterprise  simulation  to  study  risks  in  acquisition 
programs.  Then,  the  next  two  sections,  F-35  Program  Analysis  and  Risk  Analysis 
and  Mitigation,  discuss  the  application  of  enterprise  simulation  to  the  F-35  program 
and  analysis  and  mitigation  of  that  program’s  risks.  Finally,  the  last  section, 
Conclusion,  concludes  and  provides  directions  for  future  research. 

Simulation  of  Acquisition  Programs  and  Risk 

Computer  simulation  has  traditionally  been  used  to  assess  performance  of 
acquisition  and  related  processes.  One  such  widespread  use  of  simulation  has 
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been  in  manufacturing,  as  a  means  of  predicting  such  metrics  as  throughput  and 
cost  for  a  factory  (Smith,  2003).  Simulation  enables  the  analysis  of  behavior  and 
performance  overtime,  taking  into  account  probabilistic  effects  and  risk  (Law,  2007). 
Simulation  can  provide  analysis  under  different  factory  configurations  and  operating 
scenarios,  thus  enabling  possible  factory  designs,  modifications  or  operating  policies 
to  be  studied  prior  to  adoption.  Risk  can  be  studied  by  identifying  sources  of  risk 
and  modeling  their  outcomes  and  probabilities  for  occurrence. 

In  recent  years,  simulation  has  seen  application  in  a  number  of  acquisition- 
related  areas  beyond  production.  These  include  system  design  and  development 
(Bodner&  Wade,  2013),  software  system  development  (Madachy,  2008),  supply 
chain  design  and  analysis  (Kleijnen,  2005),  and  fleet  sustainment  (Smith,  Searles, 
Thompson,  &  Cranwell,  2006).  Other  work  has  used  simulation  to  study  enterprise- 
level  phenomena  in  acquisition  (Bodner,  Rahman,  &  Rouse,  2010;  Wirthlin,  2009). 
These  efforts  have  studied  a  variety  of  risk  sources,  including  mis-allocation  of 
resources,  unplanned  rework,  changing  mission  profiles,  changing  demand  patterns, 
and  poorly  designed  processes. 

The  traditional  use  of  simulation  has  been  to  study  the  technical  aspects 
associated  with  acquisition  program  elements.  These  include  work  processes, 
testing,  quality,  schedules,  part  flows,  inventory  levels,  bottlenecks,  costs,  and  lead 
times.  While  this  type  of  analysis  is  quite  useful,  it  does  not  capture  the  socio- 
technical  aspects  of  the  enterprise,  with  its  multi-actor  collaborations,  decision¬ 
making,  public-private  partnerships,  and  risks.  For  instance,  firms  that  develop  and 
manufacture  complex  products  and  systems  employ  an  enterprise  paradigm  that 
involves  large  number  of  collaborating  stakeholder  firms  (Bodner  &  Lee,  2012).  This 
type  of  enterprise  modeling  has  increasingly  been  incorporated  in  simulation  (Barjis, 
2011;  Glazner,  2011). 

Recently,  two  related  types  of  simulation  methods  have  come  into  usage  that 
address  socio-technical  aspects  of  a  system — agent-based  simulation  (Hillebrand  & 
Stender,  1994;  Saam  &  Schmidt,  2001)  and  organizational  simulation  (Nissen,  2007; 
Prietula,  Carley,  &  Gasser,  1998;  Rouse  &  Boff,  2005).  Agent-based  simulation 
focuses  on  the  interaction  of  different  agents  in  an  eco-system.  In  an  acquisition  or 
enterprise  context,  agents  can  represent  individuals  or  firms.  One  relevant  example 
of  its  use  is  in  modeling  actor  interactions  in  a  supply  chain  (Albino,  Carbonara,  & 
Giannoccaro,  2007). 

Organizational  simulation  seeks  to  model  the  behavior  of  people  and  firms  in 
the  context  of  a  world  model  and  organizational  story.  A  world  model  represents  the 
elements  internal  to  the  organization  being  modeled,  as  well  as  those  relevant 
external  elements.  The  organizational  story  represents  a  scenario  being  modeled. 
Organizational  simulation  has  been  used  to  model  healthcare  delivery,  research  & 
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development,  and  electronics  design  (Rouse  &  Bodner,  2009).  In  particular, 
research  &  development  is  characterized  by  high  levels  of  risk,  as  new  technologies 
under  development  often  do  not  work  out.  This  is  similar  to  acquisition  programs 
that  involve  development  of  new  systems,  as  opposed  to  a  new  version  of  an 
existing  system.  Thus,  one  important  use  of  simulation  is  to  determine  which 
enterprise-level  and  program-level  decisions  and  criteria  are  best-suited  to  improved 
enterprise  outcomes  in  a  risky  environment. 

Enterprise  Simulation  and  Acquisition  Risk 
Analysis 

Recent  work  has  extended  the  concept  of  organizational  simulation  to  an 
enterprise  simulation  framework  with  an  agent-based  model  for  actors  in  an 
enterprise,  a  process-oriented  model  for  phases  and  milestones  in  an  enterprise, 
and  a  decision/event  network  to  represent  a  risk-focused  way  of  representing  an 
organizational  story  (Bodner  &  Rouse,  2010).  Actors  represent  organizations  (e.g., 
firms  or  agencies)  that  participate  in  the  enterprise.  In  an  acquisition  enterprise,  they 
can  perform  a  variety  of  actions,  including 

•  communicate  with  other  actors, 

•  react  to  incentives  and  information, 

•  accrue  costs, 

•  change  variables  under  their  control,  such  as  schedule  targets  or 
budgets, 

•  progress  through  processes  and  tasks, 

•  restructure  a  program,  or 

•  terminate  a  program. 

Figure  1  shows  an  example  set  of  interactions  in  an  acquisition  context.  The 
government  program  office  interacts  with  the  lead  contractor,  as  well  as  with  other 
agencies.  The  lead  contractor  manages  the  contractor  network.  Note  that  some 
contractors  may  be  sub-contracted  to  multiple  contractors.  In  a  typical  major 
program,  of  course,  there  are  dozens  of  agencies,  some  outside  the  Department  of 
Defense  (DoD),  and  there  are  thousands  of  contractors  organized  in  a  multi-tier 
structure. 
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Figure  1 .  Example  Interactions  of  Agents  as  Firms  or  Agencies 

To  provide  a  risk-focused  organizational  story,  this  framework  utilizes  a 
concept  from  interactive  drama  and  artificial  intelligence,  that  of  a  drama  manager 
(Roberts  &  Isbell,  2008).  Interactive  drama  is  a  narrative  implemented  in 
computational  form,  similar  to  a  game,  in  which  a  user  interacts  with  a  story.  The 
drama  manager  interacts  with  the  story  flow  to  give  the  story  particular 
characteristics  (e.g.,  simplicity  versus  complexity).  It  does  this  through  a  construct 
known  as  a  plot  point  model,  which  abstracts  important  events  in  the  plot  of  the 
story.  A  plot  point  model  is  organized  as  a  directed  acyclic  graph  of  plot  point 
nodes,  some  of  which  may  have  precedence  relationships  between  them  (i.e.,  arcs), 
and  some  of  which  may  be  mutually  exclusive.  A  precedence  relationship  from  node 
1  to  node  2  (i.e.,  node  1  precedes  node  2,  or  1— >2)  implies  that  for  node  2  to  occur, 
node  1  must  occur.  Given  precedence  relationships  from  multiple  plot  points  to  a 
particular  plot  point,  only  one  of  the  predecessor  plot  points  need  occur  to  allow  the 
successor  plot  point  to  occur  (i.e.,  an  “or”  relationship  among  the  precedence 
relationships).  A  precedence  relationship  can  be  mandatory  in  that  the  predecessor 
must  occur  for  the  successor  node  to  occur  (i.e.,  enabling  “and”  relationships). 

The  graph  is  acyclic  to  disallow  two  or  more  plot  points  from  simultaneously 
having  a  cycle  of  precedence  relationships  among  them.  A  terminal  plot  point 
represents  an  end  to  the  story.  The  drama  manager  may  influence  the  plot  via 
actions  so  that  certain  events  occur  (or  are  more  likely  to  occur),  or  it  may  likewise 
prevent  certain  events  from  occurring.  The  particular  path  of  plot  points  realized  in 
an  interactive  drama  reflects  the  instantiated  plot. 

Adapted  to  a  simulation  context,  the  plot  point  model  (or  decision/event 
network)  is  used  to  represent  important  events  in  the  simulation  that  relate,  for 
example,  to  risk  in  the  unfolding  simulation.  Thus,  the  role  of  a  drama  manager  (or 
simulation  manager)  in  the  simulation  context  is  to  guide  the  simulation  via  actions 
through  a  particular  set  of  decision/event  nodes  to  reflect  a  certain  risk  profile.  The 
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outcome  of  the  simulation  can  then  be  compared  to  the  risk  profile  chosen.  Example 
simulation  manager  actions  include  the  following: 

•  Direct  selection  of  a  node  from  among  alternatives, 

•  Preference  given  to  a  particular  node  via  probabilities,  or 

•  Exclusion  of  a  particular  node. 

Figure  2  illustrates  the  relationships  among  the  simulation  manager,  the  plot 
point  model  and  the  simulation.  Certain  events  in  the  simulation  model  correspond 
to  decisions  or  events  in  the  decision/event  network.  However,  the  simulation 
contains  many  events  and  behaviors  that  are  not  critical  to  the  organizational  story 
and  hence  do  not  correspond  to  network  nodes.  Nodes  2  and  3  are  mutually 
exclusive  (only  one  at  most  may  happen).  Arcs  4— >7  and  5^7  are  mandatory 
precedence  relationships,  meaning  that  both  nodes  4  and  5  must  occur  for  7  to 
occur.  We  also  introduce  the  notion  of  both  a  trigger  node  and  an  associated 
inactive  relationship.  If  a  trigger  node  is  activated,  it  in  turn  activates  its  associated 
relationship  arc(s)  (either  a  precedence  arc  or  mutual  exclusion  arc).  For  example, 
in  Figure  2,  if  node  2  occurs,  it  may  trigger  node  A,  which  then  activates  a 
precedence  relationship  2—4. 

The  decision/event  network  prescribes  the  relationships  among  decisions  and 
events,  not  the  outcome  of  a  particular  organizational  story.  An  outcome  is  realized 
by  a  particular  sequence  of  decisions  and  events.  A  valid  sequence,  or  path,  must 
conform  to  the  precedence  and  exclusion  constraints  of  the  decision/event  network, 
and  it  can  have  only  one  terminal  node  (at  the  path  end).  Nodes  6  and  7  are  the  two 
terminal  nodes.  A  particular  decision/event  network  can  generate  many  different 
paths.  An  example  valid  path  from  Figure  2  is  1— >2— >6,  as  is  1— >3— 4— >5— >7.  Note 
that  Figure  2  does  not  specify  valid  paths,  only  precedence  relationships.  Thus, 
nodes  4  and  5  can  occur  in  a  valid  path.  In  fact,  both  must  occur  for  node  7  to 
occur. 
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Figure  2.  Simulation  and  Decision/Event  Network  Interaction 

Given  a  particular  decision/event  network,  the  enterprise  simulation 
framework  conducts  a  pre-processing  operation  to  determine  the  set  of  possible 
paths,  and  the  result  is  returned  in  the  form  of  a  partial  game  tree.  A  game  tree  is  a 
construct  from  game  theory  that  defines  moves  that  two  players  may  make  as 
alternating  nodes  in  the  tree.  The  simulation  manager  and  simulation  correspond  to 
the  two  players  in  this  context.  Nodes  in  the  decision/event  network  represent 
actions  taken  by  the  simulation  (i.e.,  decisions  and  events).  The  simulation  manager 
actions  are  not  represented  in  a  path,  as  these  actions  are  intended  to  influence  the 
path  followed  for  a  particular  simulation  outcome.  Including  the  simulation 
manager’s  possible  actions  would  make  the  full  game  tree  representation  unwieldy. 
Hence,  the  set  of  paths  or  simulation  outcomes  is  a  partial  game  tree.  The  partial 
game  tree  for  the  decision/event  network  from  Figure  2  is  shown  in  Figure  3.  The 
trigger  node  A  is  not  shown,  since  it  does  not  represent  an  event  or  decision  in  the 
simulation  other  than  to  enable  the  2— >4  precedence  relationship.  An  algorithm  for 
generating  a  partial  game  tree  from  a  decision/event  network  is  provided  in  Bodner 
and  Rouse  (2010). 
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Figure  3.  Partial  Game  Tree  From  Decision/Event  Network  in  Figure  2 

The  simulation  manager’s  actions  are  based  on  its  search  function  and 
evaluation  function.  These  functions  are  used  to  evaluate  paths  for  selection  of  a 
next  node  in  the  partial  game  tree  and  a  corresponding  simulation  manager  action. 
For  risk,  the  evaluation  can  use  the  expected  value  of  a  function  of  the  program’s 
schedule  and  cost  outcomes  for  selection  of  the  next  node.  This  must  be 
determined  over  the  set  of  path  remainders  associated  with  each  candidate’s  next 
node,  where  a  path  remainder  is  the  set  of  nodes  comprising  the  rest  of  a  particular 
path  given  a  current  node  location. 

For  example,  assume  that  the  simulation  has  executed  the  partial  path 
1  — ^3 — ^4  in  Figure  3.  Nodes  5  and  6  are  the  two  candidates  to  execute  next.  The 
possible  path  remainders  are  6,  5— >6,  and  5— >7.  The  path  with  only  node  6  is 
evaluated  using  simply  the  estimated  schedule  and  cost  values  for  node  6.  Node  5 
is  evaluated  using  a  function  of  the  estimated  schedule  and  cost  values  of  the  two 
other  path  remainders.  The  search  function  is  responsible  for  searching  through  the 
partial  game  tree  to  identify  paths  and  apply  the  evaluation  function. 

Of  course,  the  simulation  is  stochastic,  and  the  evaluation  can  only  use 
estimates  of  schedules  and  costs.  Another  issue  is  the  computational  complexity 
associated  with  the  evaluation  over  a  large  decision/event  network.  Thus,  the  look¬ 
ahead  capability  of  a  search  function  to  analyze  a  path  remainder  can  be  limited  to  a 
certain  number  of  nodes  in  the  path.  In  this  case,  an  estimate  function  must  be 
provided  for  the  last  node  to  be  considered  that  accounts  for  the  paths  that  emanate 
from  it. 

The  expectimax  formulation  (Michie,  1966)  is  one  example  of  a  search 
function  with  an  evaluation  function.  Expectimax  is  a  variant  of  minimax,  which  is  a 
formulation  used  in  two-player  games,  where  the  goal  of  one  player  (i.e.,  the 
simulation  manager)  is  to  seek  the  maximum  benefit,  while  hedging  against 
downside  outcomes  (both  measured  here  according  to  the  evaluation  function). 
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Expectimax  is  of  interest  here  because  it  specifically  includes  uncertainty 
associated  with  future  outcomes  of  the  simulation  (and  hence  future  organizational 
stories).  Pure  minimax  assumes  a  rational  opponent  in  game  play.  Expectimax, 
assuming  use  in  a  full  game  tree,  is  shown  below  in  pseudo-code: 

function  expectimax(node,  depth) 

if  node  is  a  terminal  node  or  depth  =  0 
return  the  heuristic  value  of  node 
else  if  simulation  manager  is  to  play  at  node 

//  Return  value  of  maximum-valued  child  node 
let  a  := 

foreach  child  of  node 

a  :=  max(a,  expectimax(child,  depth-1)) 
else  if  simulation  event  at  node 

//  Return  weighted  average  of  all  child  nodes’  values 
let  a  :=  0 

foreach  child  of  node 

a  :=  a  +  (Probability[child]  *  expectimax(child,  depth-1)) 

return  a 

While  the  above  function  seeks  to  maximize,  it  can  also  be  framed  as  a  minimization 
function.  Note  that  the  search  function  is  invoked  only  when  the  simulation  manager 
is  to  select  or  influence  selection  of  a  next  node  in  the  partial  game  tree.  However,  it 
executes  recursively  using  the  full  game  tree  that  consists  of  simulation  and 
simulation  manager  actions.  Typically,  at  the  simulation’s  turn  in  the  recursion,  it  is 
assumed  that  all  child  nodes  have  equal  probability  of  occurring,  in  the  absence  of 
other  information.  To  address  computational  complexity,  the  depth  of  each  search 
(i.e.,  look-ahead  capability)  may  be  bounded  (i.e.,  the  depth  parameter  in  the 
pseudo-code). 

The  interaction  of  the  simulation  manager  and  the  simulation  is  detailed 

1 .  First,  in  a  pre-processing  step,  the  simulation  manager  constructs  a 
partial  game  tree  from  the  decision/event  network.  The  decision/event 
network  is  modeler-specified.  The  partial  game  tree  is  stored. 

2.  The  simulation  manager  evaluates  each  path  in  the  partial  game  tree 
using  the  search  and  evaluation  functions. 

3.  Starting  at  the  root  node,  the  simulation  manager  performs  an  action 
using  results  from  the  search  and  evaluation  functions.  This  action 
may  be  to  select  one  of  the  root  node’s  children,  or  it  may  give 
preference  to  one  child  node,  or  it  may  exclude  one  or  more  child 
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below. 


nodes.  If  more  than  one  node  remains  feasible  after  the  simulation 
manager’s  action,  a  probabilistic  function  is  executed  to  select  the  next 
child  node. 

4.  The  simulation  then  is  initiated  to  execute  the  selected  node  and  its 
associated  events  and  behavior  not  represented  directly  in  the 
decision/event  network.  These  events  may  also  result  in  changes  to 
node  values  for  the  evaluation  function  in  the  partial  game  tree  in 
future  iterations. 

5.  The  simulation  manager  iteratively  takes  turns  with  the  simulation  as 
follows. 

a.  The  simulation  manager  marks  the  node  in  the  partial  game  tree  as 
having  occurred  and  sets  its  pointer  to  this  node  in  the  partial  game 
tree. 

b.  The  simulation  manager  invokes  the  search  and  evaluation 
functions  on  the  children  nodes  of  the  current  node. 

c.  Based  on  results  of  the  search  and  evaluation  functions,  the 
simulation  manager  performs  an  action  that  results  in  selection  of 
one  of  the  child  nodes  (similar  to  step  3  above). 

d.  Control  returns  to  the  simulation.  The  simulation  executes  the 
selected  node  and  its  associated  events  and  behavior  (similar  to 
step  4  above). 

e.  If  the  selected  node  is  a  terminal  node,  control  goes  to  step  6 
below.  Otherwise,  it  returns  to  step  5a  above. 

6.  The  simulation  ends,  and  final  statistics  are  computed. 

The  enterprise  simulation  modeling  framework  is  implemented  using 
AnyLogic™  with  a  set  of  Java  class  extensions  for  the  agent  models,  the  simulation 
manager,  plot  point  models,  and  partial  game  trees.  AnyLogic  is  a  commercially 
available  simulation  product  that  supports  multi-paradigm  modeling  using  discrete- 
event,  agent-based  and  system  dynamics  simulation. 

F-35  Program  Application 

One  major  acquisition  program  that  is  of  interest  with  respect  to  risk  analysis, 
and  application  of  the  enterprise  simulation  framework,  is  the  F-35  Joint  Strike 
Fighter  (JSF)  program.  This  section  discusses  the  F-35  program  as  an  application 
of  risk  analysis  using  enterprise  simulation. 

F-35  Program  Background 

The  F-35  program  grew  out  of  the  need  for  a  next  generation  tactical  fighter 
fleet  to  replace  the  aging  fleet  of  F-1 6s  and  F-1 8s.  Due  to  the  large  size  of  these 
legacy  fleets,  the  Joint  Strike  Fighter  effort  started  as  an  ambitious,  large-scale 
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concept.  As  the  program  became  more  defined,  Boeing  and  Lockheed  Martin 
Aeronautics  led  consortia  that  developed  concept  aircraft  in  a  competition  for  the 
system  design  and  development  contract.  When  the  Lockheed  consortium  won,  the 
Joint  Strike  Fighter  program  entered  into  system  design  and  development  (SDD)  in 
2001 .  The  initial  planned  procurement  quantity  was  2,852  planes. 

The  JSF  actually  consists  of  three  variant  aircraft  (Gertler,  2012).  The  first  (F- 
35A)  is  a  conventional  take-off  and  landing  (CTOL)  aircraft  designed  for  the  Air 
Force.  The  second  (F-35B)  is  a  short  take-off  and  vertical  landing  (STOVL)  aircraft 
designed  for  the  Marines.  The  third  (F-35C)  is  a  carrier-suitable  aircraft  (CV) 
designed  for  the  Navy. 

The  JSF  program  has  used  a  transformative  approach  to  acquisition  of  a 
major  weapons  system.  In  this  context,  transformation  means  a  major  change  in  the 
enterprise,  moving  from  an  as-is  enterprise  to  a  to-be  enterprise,  with  specific  intents 
(Kessler  &  Heath,  2006).  The  transformative  elements  of  this  approach  include  the 
following. 


•  Rather  than  the  traditional  command-and-control  relationship  between 
the  lead  contractor  and  its  suppliers,  the  contract  required  a 
partnership  model  among  three  firms — Lockheed  Martin  Aeronautics, 
BAE  Systems,  and  Northrop  Grumman  (Kessler,  McGinnis,  &  Bennett, 
2012). 

•  The  program  was  planned  as  a  global  effort,  whereby  design, 
development,  and  production  activities  would  occur  globally  across  an 
international  consortium  of  firms,  rather  than  primarily  at  one  single 
site.  Countries  with  firms  represented  in  the  consortium  would  be 
partners,  agreeing  to  purchase  F-35s  (Kapstein,  2004). 

•  Modeling  and  simulation  technologies  have  been  used  extensively  in 
SDD,  essentially  to  provide  a  “testing  in  software”  capability.  The 
confidence  in  this  testing  approach  enabled  substantial  concurrency  to 
be  designed  into  the  program  between  development  and  production  so 
that  production  started  prior  to  the  completion  of  flight  testing  (DoD, 
2011).  The  level  of  concurrency  was  higher  than  similar  programs. 

•  The  program  is  developing  three  variants  using  a  common  platform 
targeting  a  high  level  of  commonality  of  components  across  the  three 
variants.  The  intent  is  to  reduce  cost  (promote  affordability)  while 
providing  aircraft  to  support  three  mission  types.  The  three  mission 
types  require  different  high-level  technical  capabilities  (stealth,  STOVL, 
supersonic,  and  carrier  capabilities)  that  historically  have  not  been 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  10- 


combined  to  this  degree  in  a  single  craft  or  platform  (Blickstein  et  al 
2012). 

•  The  production  supply  network  was  planned  for  use  by  sustainment 
(Kessler  et  al.,  2012). 

One  of  the  first  efforts  in  the  SDD  phase  was  to  set  up  the  global  technology 
infrastructure  needed  to  support  the  design  of  the  F-35  among  the  different  firms 
involved.  This  was  a  substantive  and  transformative  initiative  that  required 
significant  alignment  among  the  enterprise  leadership  and  major  changes  to 
processes  among  all  stakeholder  firms  (Bodner  et  al.,  201 1 ).  The  technology 
requirements  were  challenging — providing  real-time  global  access  to  the  design 
database  subject  to  myriad  management  and  security  constraints.  The  technical 
challenges  turned  out  to  be  less  difficult  than  the  social  and  process  changes 
needed.  Overall,  the  set-up  of  the  technology  infrastructure  was  deemed 
successful. 

Nevertheless,  such  major  transformative  elements  entail  risk  (Rouse,  2006), 
and  there  have  been  a  number  of  less-than-desirable  outcomes  (and  potential  future 
outcomes)  for  the  program. 

•  A  reduction  in  planned  purchases  of  400  occurred  early  in  SDD,  driving 
up  unit  costs  (Blickstein  et  al.,  2012). 

•  In  the  earlier  part  of  SDD,  there  were  weight  and  design  issues  that 
required  significant  additional  effort  to  address  (Blickstein  et  al.,  2012). 

•  The  weapons  bay  of  the  CV  variant  was  redesigned  for  larger  payload, 
resulting  in  the  same  redesign  for  the  STOVL  and  CTOL  variants.  This 
redesign  caused  weight  and  stability  problems  for  the  STOVL 
(Blickstein  et  al.,  2012). 

•  Due  to  schedule  slippages  and  cost  growth,  the  program  was 
rebaselined  in  2004  and  2007  (Blickstein  et  al.,  2012). 

•  After  DoD’s  Joint  Estimating  Team  issued  a  report  in  late  2009  stating 
that  the  program  would  need  an  additional  30  months  to  complete 
SDD,  the  program  was  restructured  (Government  Accountability  Office 
[GAO],  2011),  adding  1 3  months  to  SDD  schedule  (as  well  as  the 
needed  funding),  and  withholding  $614  million  in  award  fees  from  the 
lead  contractor.  Three  aircraft  were  also  added  to  early  production. 

•  In  2010,  the  program  was  found  to  have  increased  in  cost  over  both 
the  original  baseline  and  the  then-current  baseline  by  enough  that  it 
was  certified  with  a  Nunn-McCurdy  breach  (Blickstein  et  al.,  2012). 
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•  Continued  technical  issues  with  the  STOVL  variant  resulted  in  that  part 
of  the  program  being  placed  on  a  two-year  “probation,”  with  its 
production  schedule  moved  back  (GAO,  2011). 

•  The  U.S.  government’s  fiscal  situation  has  called  into  question  whether 
the  planned  quantities  of  aircraft  will  be  purchased.  Already  the  UK 
government  has  reduced  its  planned  purchases  (Gertler,  2012). 

Blickstein  et  al.  (2012)  conducted  a  root  cause  analysis  of  the  Nunn-McCurdy 
breach  and  identified  the  following  underlying  risk  factors  as  root  causes: 

•  SDD  was  designed  around  a  concept  demonstrator  rather  than  around 
a  prototype  that  would  have  accounted  better  for  producibility.  Harvey 
and  Ryan  (2012)  found  evidence  that  a  prototype  can  reduce  cost 
growth  in  procurement  for  fixed-wing  aircraft. 

•  The  engineering  strategy  focused  on  developing  the  CTOL  variant  first, 
rather  than  tackling  the  more  difficult  engineering  design  and 
development  of  the  STOVL. 

•  The  original  baseline  was  overly  optimistic  on  cost  and  schedule  due  to 
assumptions  on  component  commonality  and  technology  integration. 

These  in  turn  drove  other  issues  that  resulted  in  the  eventual  breach.  Getting  the 
high-level  technical  capabilities  across  the  variants  was  more  technically  challenging 
than  anticipated,  involving  extensive  trade-offs.  The  initial  weight  problems  caused 
redesign,  and  these  design  changes  had  to  be  distributed  throughout  the  enterprise 
of  collaborators.  Due  to  the  large  number  of  suppliers,  this  process  delayed  test  and 
production  and  hence  drove  up  costs.  Delays  also  resulted  in  work  being  done  with 
higher  labor  rates. 

The  issue  of  concurrency  has  been  raised  as  a  cause  of  the  program’s  cost 
and  schedule  issues  (DoD,  201 1 ).  Historical  analysis  has  indicated  that  concurrency 
has  not  been  a  major  cost  and  schedule  risk  driver  (Congressional  Budget  Office 
[CBO],  1988).  However,  this  type  of  analysis  likely  does  not  account  for  the 
complexity  of  the  JSF  program. 

At  present,  the  program  is  entering  a  fifth  low-rate  initial  production  (LRIP) 
increment  that  is  a  fixed-price  contract  with  risk  sharing  between  the  government 
and  contractor  (Gertler,  2012). 

Decision/Event  Network 

The  decision/event  model  for  the  model  of  the  JSF  SDD  is  shown  below  in 
Figure  4,  and  detailed  descriptions  for  each  node  are  provided  in  Table  1 .  At  the  top 
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of  the  figure  are  the  major  program  design  decisions  modeled  in  this  example. 

These  include 

•  whether  a  prototype  article  is  developed  prior  to  full-scale  development 
(i.e.,  beyond  the  concept  demonstrator  that  was  developed  prior  to 
SDD), 

•  whether  the  CTOL  variant  or  the  STOVL  variant  is  given  first  design 
priority, 

•  whether  the  program  engages  in  high  concurrency  or  no  concurrency, 
and 

•  whether  the  program  engages  in  an  alignment  effort  at  the 
commencement  of  SDD. 

These  are  risk  drivers  to  be  explored  as  the  JSF  program  unfolds. 
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Figure  4.  JSF  SDD  Decision/Event  Network 

If  a  prototype  is  developed,  it  is  assumed  that  a  weight  reduction  program 
later  in  development  is  not  needed.  Thus,  nodes  2  and  1 1  are  mutually  exclusive.  If 
a  CTOL-first  design  and  development  strategy  is  adopted,  then  trigger  node  A 
causes  the  CDR  for  the  CTOL  variant  to  occur  before  that  of  the  STOVL,  as  well  as 
the  flight  test  for  the  CTOL  variant  to  occur  before  that  of  the  STOVL.  Similarly,  if  a 
STOVL-first  design  and  development  strategy  is  adopted,  the  converse  occurs. 
There  are  two  aggregating  nodes  without  labels  that  are  used  to  represent  that  one 
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of  the  CTOL  or  STOVL  variant  first  design  strategies  must  occur,  as  well  as  one  of 
the  concurrency  alternatives. 

After  program  design,  the  program  must  attain  global  design  capability  to 
enable  collaborating  firms  to  co-design  the  F-35.  There  may  or  may  not  be  an  effort 
by  the  leaderships  of  the  top-tier  partner  firms  to  achieve  alignment  prior  to  this. 

After  global  design  capability  is  achieved,  preliminary  design  review  (PDR)  can 
occur,  and  then  critical  design  review  (CDR)  and  flight  testing.  Note  that  CDR  and 
flight  testing  are  separate  for  each  variant.  Also,  it  is  assumed  for  this  analysis  that 
the  CV  variant  undergoes  both  CDR  and  flight  testing  after  the  other  two  variants 
have  done  so.  Thus,  if  the  no-concurrency  program  design  decision  is  made, 
inactive  arc  C  is  activated,  causing  LRIP  not  to  occur  before  all  flight  testing  is  done. 

The  two  terminal  nodes  represent  either  full-rate  production  at  the  planned 
level  of  production,  or  a  50%  reduction  that  results  from  excessive  cost  growth,  in  an 
effort  to  reduce  the  overall  program  cost. 


Ta 

blel.  Decision-Event  Descriptions 

Decision-Event  Node 

Description 

1 .  Contract  Awarded 

SDD  contract  awarded  to  Lockheed  Martin  consortium.  This 
initiates  the  organizational  story. 

2.  Prototype  Development 

This  represents  the  program  enterprise  decision  to  develop  or 
not  develop  a  prototype  aircraft  in  addition  to  the  concept 
aircraft  that  won  the  SDD  contract.  Note  that  an  affirmative 
decision  delays  PDR. 

3.  CTOL-First  Strategy 

The  program  decides  to  design  the  CTOL  variant  first. 

4.  STOVL-First  Strategy 

The  program  decides  to  design  the  STOVL  variant  first.  Note 
that  this  is  mutually  exclusive  with  the  CTOL-first  strategy. 

5.  High  Concurrency 

A  high  level  of  concurrency  is  designed  into  the  program. 

6.  No  Concurrency 

No  concurrency  is  designed  into  the  program.  Note  that  this  is 
mutually  exclusive  with  the  high  concurrency  decision. 

7.  Program  Design 

Complete 

Program  design  is  complete. 

8.  Collaborative  Alignment 

The  enterprise  conducts  an  effort  to  achieve  collaborative 
alignment  on  the  changes  needed  to  enable  the  global  SDD 
technology  infrastructure  to  be  stood  up.  Achievement 
includes  executive  alignment  among  partner  organizations,  as 
well  as  technical  management  alignment.  Achievement 
speeds  up  technology  development  over  the  case  where  there 
is  little  alignment. 

9.  Global  Design 

Capability 

The  enterprise  achieves  capability  to  support  JSF  design  via 
global  technology  infrastructure. 

10.  PDR 

The  program  passes  preliminary  design  review. 

11.  Weight  Reduction 

The  program  engages  in  a  major  weight  reduction  effort. 

12.  CTOL  CDR 

The  CTOL  variant  passes  critical  design  review.  Note  that 
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CDR  is  a  separate  event  for  each  variant. 

13.  STOVL  CDR 

The  STOVL  variant  passes  critical  design  review. 

14.  CV  CDR 

The  CV  variant  passes  critical  design  review. 

15.  CTOL  Flight  Test  Start 

The  CTOL  variant  starts  flight  testing.  Once  again,  flight  tests 
are  modeled  separately  for  each  variant. 

16.  STOVL  Flight  Test 

Start 

The  STOVL  variant  starts  flight  testing. 

17.  CV  Flight  Test  Start 

The  CV  variant  starts  flight  testing. 

18.  LRIP 

Low  rate  initial  production  begins. 

19.  Production  Ramp-up 

Production  ramp-up  begins.  This  is  a  terminal  decision/event. 

20.  Major  Planned  U.S. 
Purchase  Reduction 

The  U.S.  decides  to  reduce  its  purchase  quantity  by  50%. 

A  relatively  simple  agent  interaction  model  is  used  to  model  the  actors  in  the 
program.  The  program  office,  the  lead  contractor,  and  partner  contractors  are 
agents  that  interact  in  terms  of  cost  accruals,  program  decisions,  and  program 
alignment. 

Baseline  Model — Variables,  Parameters,  and  Model  Logic 

The  primary  variables  relate  to  schedule  and  cost.  Supporting  data  is  derived 
from  the  2012  Selected  Acquisition  Report  (SAR)  for  the  F-35  (DoD,  2012).  Cost 
variables  are  consistent  with  the  2012  SAR  in  that  cost  is  divided  into  the  categories 
of 


•  research,  development,  testing,  and  evaluation  (RDT&E), 

•  procurement,  and 

•  military  construction. 

The  SAR  further  subdivides  costs  between  the  Navy  (STOVL  and  CV  variants)  and 
the  Air  Force  (CTOL  variant)  and  also  considers  the  aircraft  costs  separately  from 
the  engine  costs,  since  the  engine  was  categorized  as  a  separate  program.  In  the 
model,  Navy  and  Air  Force  costs  and  the  aircraft  and  engine  costs  are  combined 
under  the  three  main  categories  of  RDT&E,  procurement,  and  military  construction. 

It  should  be  noted  that  the  RDT&E  costs  include  non-Treasury  funds  (i.e.,  costs  paid 
by  partner  countries  in  the  F-35  program).  In  addition,  base  year  2012  dollars  are 
used  to  track  costs,  providing  a  consistent  unit  of  measure  for  the  span  of  the 
program.  It  should  be  noted  that  the  use  of  base  year  dollars  has  been  criticized  as 
a  unit  of  measure  because  it  understates  actual  out-year  costs  due  to  the  effect  of 
inflation.  The  main  variables  are  as  follows: 

7,  =  occurrence  time  for  node  /  in  the  decision/event  network  (in  months) 
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Rt  =  cost  accrual  of  RDT&E  expenditures  as  of  month  t  (in  base  year  2012 
dollars) 

Pt  =  cost  accrual  of  procurement  expenditures  as  of  month  t  (in  base  year  2012 
dollars) 

Ut  =  estimated  final  unit  cost  as  of  month  t  (in  base  year  2012  dollars  per  aircraft 
unit) 

Unit  cost  is  computed  as  Program  Acquisition  Unit  Cost  (PAUC).  PAUC 
considers  RDT&E  costs,  procurement  costs,  and  military  construction  costs  in  the 
overall  program  cost,  which  is  then  divided  by  the  number  of  units  to  be  made  for 
both  RDT&E  (i.e.,  test  articles)  and  procurement.  This  unit  cost  measure  is  used  in 
preference  to  Average  Procurement  Unit  Cost  or  fly-away  cost,  since  PAUC 
considers  RDT&E  and  military  construction  expenditures.  This  is  especially  relevant 
to  a  program  such  as  the  F-35  that  is  non-derivative  of  prior  programs  and  thus  has 
a  substantial  RDT&E  cost  component.  PAUC  not  only  includes  those  costs  currently 
accrued,  but  also  costs  projected  through  the  end  of  full  rate  production  in  2037. 
Thus,  in  the  simulation  model,  Ut  is  computed  via  a  look-ahead  replication  of  the 
simulation  starting  at  t  and  ending  at  production  completion.  This  replication  factors 
in  the  probability  of  full  production  (decision/event  node  19)  versus  reduced 
purchase  (decision/event  node  20). 

Cost  accruals  for  RDT&E  and  procurement  are  assumed  to  occur  at  a  rate 
dependent  on  the  current  phase  of  the  program.  These  rates  are  parameters  in  the 
simulation  model  and  are  computed  using  cost  data  from  the  2012  SAR  (in  base- 
year  2012  dollars).  Tables  2  and  3  show  the  monthly  spend  rates  for  both  RDT&E 
and  procurement  and  the  time  periods  from  which  they  are  derived  using  the  SAR 
data.  These  are  computed  by  converting  the  annual  SAR  data  to  monthly  rates.  In 
cases  where  the  period  spans  only  a  partial  year,  the  annual  rate  is  pro-rated  and 
then  converted  to  monthly.  For  instance,  an  18-month  period  spanning  a  full  year 
and  half  of  another  would  add  the  first  year’s  cost  plus  half  the  second  year’s  cost, 
then  divide  by  the  18  months  to  obtain  a  monthly  rate.  In  the  simulation  model,  the 
rates  are  random  variables  uniformly  distributed  around  the  parameters  in  Tables  2 
and  3  via  the  distributional  form  U(0.95c,  1 ,05c),  where  c  is  the  cost  rate  in  question. 
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Table  2.  Spend  Rates  for  RDT&E 


Program  Phase 

Rate 

Period  of  Derivation 

Initial  accrual  prior  to  SDD 

$4,495M  total 

1994-9/2001 

Up  to  PDR 

$245. 5M  per  month 

10/2001-4/2003 

PDRtoCDR 

$448. 9M  per  month 

5/2003-6/2007 

Flight  test — segment  1 

$380. 6M  per  month 

7/2007-12/2008 

Flight  test — segment  2 

$332. 8M  per  month 

1/2009-6/2010 

Flight  test — segment  3 

$141. 7M  per  month 

7/2010-12/2018 

Table  3.  Spend  Rates  for  Procurement 


Program  Phase 

Rate 

Period  of  Derivation 

Initial  accrual  prior  to  SDD 

$0  total 

1994-9/2001 

LRIP  preparation 

$47. 9M  per  month 

3/2006-3/2007 

LRIP 

$597. 7M  per  month 

4/2007-9/2018 

Full  production  (assuming 
all  planned  aircraft  are 
produced) 

$178, 801. 4M  total 

10/2018-9/2037 

Two  additional  parameters  are  the  total  amount  of  military  construction 
expenditures  M  ($3,897.8  million)  and  the  total  number  of  aircraft  that  are  planned 
for  production  (2,457,  14  of  which  are  test  planes  produced  via  RDT&E). 

The  decision  as  to  whether  the  program  performs  full  rate  production  at  the 
planned  level  or  whether  it  reduces  the  buy  by  50%  is  made  probabilistically 
according  to  the  following  functions  in  Equations  1  and  2: 

f  1  if  Ut19  <  A 

Pfproduce  planned  amount )  =  <  (B  —  UTl9)/ (B  —  A)  if  A  <  UTi9  <  B 

{  0  if  uTi9  if  >b 

Pfreduce  buy  by  50%)  —  1  —  P (produce  planned  amount ) 

Recall  that  UTi9  is  the  projected  unit  cost  (PAUC)  at  the  end  of  LRIP, 
assuming  that  full  rate  production  is  to  occur.  Thus,  if  UTi9  is  less  than  a  threshold  A, 
then  full  rate  production  occurs.  If  it  is  over  a  threshold  B,  then  the  full  buy  does  not 
occur.  Otherwise,  there  is  a  probability  that  decreases  linearly  with  UTi9  whereby  full 
rate  production  occurs.  For  the  baseline  model,  A  is  set  to  $80  million,  while  B  is  set 
to  $200  million.  This  corresponds  to  program  costs  of  $1 96.6  billion  and  $491 .4 
billion,  respectively,  for  the  full  purchase. 

Of  course,  if  the  purchase  amount  is  reduced  by  50%,  this  dramatically 
affects  the  unit  cost,  since  the  fixed  RDT&E  and  military  construction  costs  must  be 
amortized  over  a  smaller  number  of  units.  Also,  the  procurement  cost  is  not  strictly 
linear,  since  efficiencies  of  increasing  returns  to  scale  are  typically  built  into 
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projected  production  costs.  Thus,  one  cannot  simply  reduce  the  procurement  cost 
by  half  and  obtain  the  procurement  cost  for  50%  fewer  planes. 

In  theory,  these  returns  to  scale  are  modeled  by  such  approaches  as  a  Cobb- 
Douglas  production  function  (Cobb  &  Douglas,  1928),  which  uses  an  exponential 
function  to  relate  inputs  to  outputs  such  as  production  quantity.  Depending  on  the 
sign  of  the  exponents,  different  returns  to  scale  behaviors  can  be  represented, 
including  increasing,  decreasing,  or  constant.  However,  the  SAR  data  are  not 
amenable  to  a  statistical  fit  of  such  an  exponential  function  using  cost  as  input  and 
production  quantity  as  output.  Thus,  a  piece-wise  linear  function  was  constructed  by 
linear  regression.  Experimentation  was  performed  to  determine  best  cut-off  points 
between  a  relatively  small  number  of  different  linear  segments,  based  on  the  best 
statistical  fit.  The  results  are  shown  in  Table  4,  where  Q  is  the  production  quantity, 
and  P  is  the  procurement  cost.  Note  that  the  production  quantity  funded  by 
procurement  is  2,443,  whereas  RDT&E  is  used  to  fund  14  additional  aircraft.  Also, 
these  equations  do  not  apply  for  small  quantities  of  aircraft. 


)le  4.  Regression  Results 

or  Production  Function  of 

F-35  Procure 

Quantity  (Q) 

Years 

Production  Equation 

R2 

0-179 

2007-14 

Q  =  -4.76  +  0.00455P 

0.999 

180-559 

2015-19 

Q  =  -182  +  0.00836P 

0.994 

560-1,960 

2020-21 

Q  =  -427  +  0.011  OP 

1.000 

1,961-2,443 

2032-37 

Q  =  -712  +  0.01 23P 

1.000 

If  a  purchase  reduction  is  made,  it  reduces  the  total  purchased  from  2,443  to 
1 ,221 .  This  quantity  falls  into  the  third  row  of  Table  4,  with  an  input  procurement 
cost  of  P  =  $149, 818. 2M.  The  14  aircraft  purchased  from  RDT&E  are  included, 
then,  in  the  total  program  quantity  of  1 ,235  considered  for  the  PAUC  computation. 
The  following  two  parameters  are  defined: 

u-\9  =  production  quantity  for  full  production  (node  19)  =  2,457 
u2Q  =  production  quantity  for  reduced  purchase  (node  20)  =  1 ,235 

The  initial  month  is  set  to  the  kick-off  of  the  SDD,  which  is  October  of  2001 . 
Thus,  t  =  1  refers  to  this  month,  and  7i  =  1  (i.e.,  contract  award  is  defined  to  occur  in 
the  first  month  of  the  simulation). 

Validation 

The  baseline  model  is  validated  against  current  outcomes  from  the  F-35 
program  by  using  the  path  in  the  decision/event  network  that  corresponds  to  the 
existing  program,  as  shown  in  Figure  6.  Note  that  this  assumes  full  production,  as 
does  the  current  program. 
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Figure  5.  Current  JSF  Program 

Comparisons  from  25  simulation  replications  against  outcomes  from  the 
existing  JSF  program  are  shown  in  Table  5.  Existing  program  figures  come  from  the 
2012  SAR.  It  should  be  noted  that  these  existing  figures  include  projections  from 
present  until  the  expected  initiation  of  full  rate  production  in  September  2018. 


Table  5.  Validation 


Outcome 

Actual/Projected 
Program  Outcome 

95%  Confidence  Interval 

From  Simulation  Outcomes 

Expected  schedule  at  initiation 
of  FRP  (months) 

204  (Sept  2018) 

(202.5,  206.0) 

RDT&E  cost  incurred  at 
initiation  of  FRP  (base  year 
2012$) 

$59,398.1  M 

($58,31 7. 7M,  $59,450.4M) 

Procurement  cost  at  initiation 
of  FRP  (base  year  2012  $) 

$77,328.7M 

($81 ,602.3M,  $83,647.4M) 

Unit  cost  (PAUC,  base  year 
2012  $  per  unit) 

$130. 0M 

($131.0,  132.7M) 

The  values  from  the  baseline  simulation  are  reasonably  in  line  with  the 
program  outcome  values.  The  schedule  and  the  RDT&E  costs  incurred  lie  within 
their  respective  95%  confidence  intervals.  The  other  figures  from  the  simulation  are 
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biased  upward  slightly  via  parameters  to  account  for  a  greater  chance  that  the  actual 
program  outcomes  will  be  higher  rather  than  lower  than  the  ones  projected  from  the 
2012  SAR.  One  primary  reason  is  that  the  procurement  cost  is  based  on  SAR  data 
that  includes  a  five-year  projection  of  outcomes  through  2018.  The  simulated 
outcome  of  unit  cost  is  biased  upward  since  it  not  only  includes  the  projection  of 
procurement  costs  through  2018,  but  also  procurement  costs  through  2037  (the 
anticipated  end  of  full  rate  production).  RDT&E,  on  the  other  hand,  is  mostly 
expended  by  this  point  in  the  program,  and  final  expenditures  are  more  certain. 

Alternate  Scenarios 

The  baseline  simulation  model  is  enhanced  with  additional  scenarios  for 
analysis,  which  include  a  possible  prototype,  a  STOVL-first  design  strategy,  a  no¬ 
concurrency  option,  and  the  lack  of  an  effort  at  alignment  in  preparation  of  global 
design.  Altogether,  32  different  paths  are  considered  in  the  partial  game  tree,  shown 
in  Table  6.  Path  17  refers  to  the  baseline  case.  Path  18  assumes  outcomes  of  the 
current  program,  with  reduced  purchase  in  the  future. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


■21 


Table  6.  Paths  Considered  in  Partial  Game  Tree 


Prototype1 

CN 

-4— ' 

c 

CD 

L. 

CD 

> 

to 

Concurrency3 

Alignment4 

Outcome5 

Path  (using  nodes  as  defined  in  Table  1) 

1 

Y 

C 

H 

Y 

F 

1  ->2->3->5->7->8^9-^  1 0^  1 2^  1 3-*  1 4->  1 5-*  1 8^  1 6->  1 7->  1 9 

2 

Y 

c 

H 

Y 

R 

1  -^2->3->5-+7->8^9^  1 0->  1 2->  1 3->  1 4->  1 5-> 1 8^  1 6->  1 7->20 

3 

Y 

c 

H 

N 

F 

1  ->2->3— >5— >7— >9— >10— >12— >13— >14— >15— >18— >16— >17— >19 

4 

Y 

c 

H 

N 

R 

1  — >2— >3->5— >7— >9— >1 0— >1 2— >1 3->14— >1 5->1 8— >1 6->1 7->20 

5 

Y 

c 

N 

Y 

F 

1  ->2->3->6->7->8^9-^  1 0^  1 2->  1 3->  1 4^  1 5->  1 6->  1 7^  1 8^  1 9 

6 

Y 

c 

N 

Y 

R 

1  ^2->3->6->7->8^9^  1 0^  1 2-+ 1 3->  1 4->  1 5->  1 6->  1 7->  1 8^20 

7 

Y 

c 

N 

N 

F 

1  — >2 — >3 — >6 — >7 — >9 — >1 0 — >12 — >1 3 — >14 — >1 5 — >1 6 — >1 7 — >1 8 — >1 9 

8 

Y 

c 

N 

N 

R 

1  — >2— >3->6->7— >9— >1 0— >1 2— >1 3->1 4— >1 5->1 6— >1 7->1 8— >20 

9 

Y 

s 

H 

Y 

F 

1  -+2^4->5->7->8^9-^  1 0^  1 3->  1 2->  1 4->  1 6->  1 8^  1 5->  1 7-+ 1 9 

10 

Y 

s 

H 

Y 

R 

1  ^2->4->5-V7->8^9^  1 0^  1 3->  1 2->  1 4-+ 1 6->  1 8^  1 5->  1 7-+20 

11 

Y 

s 

H 

N 

F 

1->2^4->5-V7->9-+10^13->12^14-+16->18^15^17-+19 

12 

Y 

s 

H 

N 

R 

1  — >2— >4->5— >7— >9— >10— >13— >12->14— >16->18— >15->17->20 

13 

Y 

s 

N 

Y 

F 

1  ->2->4->6->7->8^9-^  1 0^  1 3->  1 2^  1 4->  1 6->  1 5->  1 7->  1 8^  1 9 

14 

Y 

s 

N 

Y 

R 

1  ^2^4->6->7->8^9^  1 0^  1 3-+ 1 2->  1 4-+ 1 6->  1 5->  1 7->  1 8^20 

15 

Y 

s 

N 

N 

F 

1  — >2 — >4 — >6 — >7 — >9 — >1 0 — >1 3 — >12 — >14 — >1 6 — >1 5 — >1 7 — >1 8 — >1 9 

16 

Y 

s 

N 

N 

R 

1  — >2— >4— >6— >7— >9— >1 0— >1 3— >1 2->1 4— >1 6->1 5— >1 7->1 8— >20 

17 

N 

c 

H 

Y 

F 

1  — >3— >5->7— >8— >9— >1 0— >1 1  ->1 2^1 3-+1 4->1 5->1 8^1 6-+1 7->1 9 

18 

N 

c 

H 

Y 

R 

1  ->3— >5— >7— >8— >9— >10— >1 1— >12— >13-^14-^15— >18^16-^17— >20 

19 

N 

c 

H 

N 

F 

1 ^3^5>7>9^  1 0^  1 1 ->  1 2->  1 3-^  1 4^  1 5->  1 8^  1 6-^  1 7->  1 9 

20 

N 

c 

H 

N 

R 

1  ->3-+5->7->9-+ 1 0^  1 1  ->  1 2->  1 3->  1 4->  1 5->  1 8^  1 6->  1 7->20 

21 

N 

c 

N 

Y 

F 

1  — >3— >6->7->8— >9— >1 0— >1 1  —>12—^1 3— ^14— >1 5— >1 6— >  1 7— >  1 8— »•  1 9 

22 

N 

c 

N 

Y 

R 

1  — >3— >6— >7— >8— >9— >1 0— >1 1  ->1 2->1 3->1 4->1 5->1 6->1 7->1 8^20 

23 

N 

c 

N 

N 

F 

1  — >3 — >6 — >7 — >9 — >1 0 — >1 1  — >-12 — >-1 3 — >14 — >1 5 — >16 — >1 7 — >1 8 — >1 9 

24 

N 

c 

N 

N 

R 

1  ->3^6->7->9-^  1 0^  1 1  ->  1 2->  1 3->  1 4->  1 5->  1 6->  1 7->  1 8^20 

25 

N 

s 

H 

Y 

F 

1  — >4— >5->7— >8— >9— >1 0— >1 1  —>1 3— >12— ►14— >1 6— >1 8—^1 5— ^1 7— >1 9 

26 

N 

s 

H 

Y 

R 

1  ->4— >5— >7— >8— >9— >1 0— >1 1  ->1 3— >12— *-14— +1 6->1 8->1 5->1 7->20 

27 

N 

s 

H 

N 

F 

1  ->4-^5— >7— >9-^  1 0— >  1 1  — >  1 3— >  1 2— >  1 4-^  1 6— >  1 8^  1 5-^  1 7— >  1 9 

28 

N 

s 

H 

N 

R 

1  — >4— >5— >7— >9— >1 0— >1 1  — >-1 3 — >12 — >14 — >1 6 — >1 8 — >1 5 — >1 7 — >20 

29 

N 

s 

N 

Y 

F 

1  — >4— >6->7->8— >9— >1 0— >1 1  ->1 3->1 2->1 4->1 6->1 5->1 7->1 8->1 9 

30 

N 

s 

N 

Y 

R 

1  — >4— >6— >7— >8— >9— >1 0— >1 1  ->1 3— *-12— »14— +1 6->1 5->1 7->1 8^20 

31 

N 

s 

N 

N 

F 

1  —>4—^6— >7— >9-^  1 0— >  1 1  — >  1 3— >  1 2^  1 4^  1 6— >  1 5— >  1 7— >  1 8^  1 9 

32 

N 

s 

N 

N 

R 

1  -^4->6->7->9-^  1 0^  1 1  ->  1 3->  1 2->  1 4->  1 6->  1 5->  1 7->  1 8^20 

1  “Y”  means  prototype  developed;  “N”  means  no  prototype. 

2  “C”  means  CTOL-first  design;  “S”  means  STOVL-first. 

3  “H”  means  high  concurrency;  “N”  means  no  concurrency. 

4  “Y”  means  alignment  sought;  “N”  means  no  alignment  sought. 

5  “F”  means  full  production  of  planned  quantity;  “R”  means  reduction  of  50%. 

Risk  Analysis  and  Mitigation 

This  section  describes  use  of  the  simulation  model  to  perform  risk  analysis. 
The  simulation  produces  performance  measures  of  unit  cost  (PAUC),  schedule, 
RDT&E  cost,  and  procurement  cost.  Unit  cost  is  the  primary  performance  measure 
chosen  for  analysis  and  is  the  decision  criterion,  since  it  is  one  of  the  prime 
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measures  used  to  judge  program  success  and  program  cost  breaches.  The  other 
measures  will  be  examined,  as  well. 

Risk  Perspectives 

Two  different  scenarios  are  studied,  consisting  of  differing  parameters.  Within 
each  case,  three  perspectives  are  examined.  The  first  perspective  is  that  of  a  risk- 
neutral  decision-maker.  A  risk-neutral  decision-maker  is  defined  as  one  that  would 
prefer  to  minimize  the  expected  unit  cost  for  the  F-35  acquisition.  The  second 
perspective  is  that  of  a  risk-averse  decision-maker.  A  risk-averse  decision-maker 
would  like  to  minimize  the  maximum  expected  unit  cost.  The  final  perspective  is  that 
of  a  risk-seeking  decision-maker.  A  risk-seeking  decision-maker  seeks  to  minimize 
the  minimum  expected  unit  cost. 

The  expectimax  framework  is  used  for  the  search  and  evaluation  functions  for 
each  of  these  three  perspectives.  The  objective  is  to  minimize  some  function  of  unit 
cost.  The  search  and  evaluation  functions  operate  as  follows. 

Let  node  /  be  a  node  in  the  partial  game  tree,  and  let  C,  be  the  set  of  child 
nodes  of  /.  Let  /*  be  the  number  of  the  node  in  the  decision/event  network  that 
corresponds  to  node  /.  Note  that  the  partial  game  tree  may  contain  multiple  copies 
of  each  node  in  the  decision/event  network.  The  node  index  in  the  decision  event 
network  is  used  to  reference  simulation  output  variables  to  the  corresponding  node 
in  the  partial  game  tree. 

Let  E(i)  be  the  evaluation  function  for  /.  Let  S,  be  the  set  of  terminal  nodes  for 
the  set  of  paths  that  emanate  from  /.  For  j  e  S,,  E(j)  is  defined  as  a  function  of  the 
final  projected  unit  cost  at  the  time  of  node  j. 

£(y)  =  f(Urt,.j )  (3) 


Note  that  the  projected  unit  cost  estimate  is  made  at  the  time  in  the  simulation 
of  node  /.  Also,  the  function  is  dependent  on  the  particular  node  j.  In  the  F-35 
example,  for  instance,  the  unit  cost  of  the  terminal  node  depends  on  whether  there  is 
full  production  or  reduced  purchase. 

For  j  such  that  for  all  k  e  Cy,  k  e  S„  E(j)  is  defined  as  the  expected  value  of  the 
unit  cost  over  the  child  nodes  weighted  by  the  probabilities  in  Equations  1  and  2. 

For  Equation  3,  pi9  is  given  by  Equation  1 ,  and  p2 o  is  given  by  Equation  2,  where 
nodes  19  and  20  are  the  two  terminal  nodes. 

£(/)  —  TjkeCiPk*E(k)  (4) 

For  j  such  that  for  all  k  e  Cy,  k  £  Si,  E(j)  is  defined  as  the  expected  value  of  the 
unit  cost  averaged  over  the  child  nodes.  Note  that  E(i)  can  be  computed  with 
Equations  3  or  4. 
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£(/)  =  Efcec£^£(/c) 
iwi 


The  search  function  is  exhaustive,  meaning  that  it  examines  each  path 
emanating  from  /  to  its  end.  It  does  this  by  executing  a  simulation  for  each  path 
starting  at  node  /  and  terminating  at  the  path’s  terminal  node  and  using  Equations  3 
to  5  to  provide  an  evaluation  function  for  the  children  of  node  /.  Note  that  the 
function  form  for  f(UT.t,j )  depends  on  the  risk  perspective  being  utilized.  For  each 
case,  the  function  utilizes  the  current  RDT&E  and  procurement  cost  accruals  (RT.t 
and  PT.  )  as  the  starting  point,  and  the  projected  function  of  final  unit  cost  is 
computed  by  the  simulation. 


For  the  risk-neutral  perspective,  the  functional  form  sums  military 
construction,  current  cost  accruals,  and  an  average  value  for  future  cost  accruals, 
namely  Rexp{Ti*^  and  Pexp(Ti *),  which  are  computed  with  a  simulation  replication 
using  distributional  averages  of  cost  and  schedule  variables  in  the  simulation.  It 
then  divides  by  the  units  produced  at  terminal  node  j. 

f.^jj  j}  _  M+RTi*+PTi*+Rexp(Ti*^+Pexp(Ti*^ 


Similarly,  the  risk-averse  perspective  substitutes  Rmax(Ji *)  and  Pmax(Ti*)  for 
the  value  of  future  cost  accruals.  These  terms  are  computed  using  a  simulation 
replication  from  using  distributional  maximums  for  cost  and  schedule  variables. 
Finally,  the  risk-seeking  perspective  substitutes  Rmin(JiJ  and  Pmin(Tiif )  for  the  value 
of  future  cost  accruals.  These  terms  are  computed  with  a  simulation  replication 
using  distributional  minimums  for  cost  and  schedule  variables. 

For  each  perspective  within  each  case,  100  replications  of  the  simulation  are 
made.  The  output  results  are  then  analyzed  using  Minitab  16. 

Scenario  1 

For  the  different  risk  perspectives  in  the  first  scenario,  the  following 
assumptions  are  made  and  parameters  adopted  using  the  baseline  model  as  a 
starting  point. 

For  the  case  of  no  alignment  sought,  the  attainment  of  global  design 
capability  is  pushed  back  by  three  months. 

For  the  case  of  no  concurrency,  LRIP  commences  after  flight  testing 
begins  for  the  C V  variant  by  a  period  of  months  triangularly  distributed 
~TR(16,  21,  18).  In  this  notation,  the  first  parameter  is  the  minimum, 
the  second  the  maximum  and  the  third  the  mode.  The  triangular 
distribution  is  used  often  in  simulation  when  there  is  little  information 
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on  an  exact  distributional  form.  LRIP  lasts  for  a  period  of  months 
distributed  ~TR(60,  75,  68)  months.  However,  LRIP  is  given  a  75% 
increase  in  procurement  cost  rate  due  to  the  focused  effort  in  a 
shortened  timeframe  separate  from  design. 

•  For  the  STOVL-first  design  case,  the  STOVL  CDR  occurs  after  the 
CTOL  CDR  design  would  have  occurred  by  a  period  of  months 
distributed  ~TR(6,  9,  27),  allowing  additional  time  for  the  more  complex 
requirements  for  the  STOVL.  However,  the  flight  testing  begins  twelve 
months  sooner. 

•  For  the  prototype  development  case,  PDR  is  pushed  back  an  amount 
of  time  distributed  ~TR(20,36,24),  and  the  PDR  cost  rate  is  increased 
by  50%  to  reflect  the  prototype  development  cost.  Both  CDR  and  full 
rate  production  are  pushed  forward  by  a  period  of  months  distributed 
~TR(9,18,15),  reflecting  efficiencies  gained  by  knowledge  from  the 
prototype.  Finally,  the  procurement  cost  rate  is  decreased  by  10%  to 
allow  for  knowledge  gained  by  the  prototype,  as  an  estimate  reflecting 
results  from  Harvey  and  Ryan  (2012)  indicating  that  procurement  costs 
can  be  reduced  by  a  prototype. 

•  For  the  case  of  prototype  development  and  no  concurrency,  full-rate 
production  is  pushed  forward  by  a  period  of  months  distributed  ~TR(3, 
10,  8)  instead  of  ~TR(9,18,15),  reflecting  increased  schedule  needed 
for  no  concurrency. 

Summary  performance  statistics  from  the  100  simulation  replications  for  the 
three  perspectives  are  shown  below  in  Table  7.  The  column  with  heading  “Full 
Production”  denotes  the  percentage  of  simulation  outputs  that  result  in  full 
production  versus  reduced  purchase. 


Table  7. 

Average  Simulation  Results  for  Each 

Perspective  in  Scenario  1 

Case 

Unit  Cost 

Schedule 

RDT&E  Cost 

Proc.  Cost 

Full  Production 

Risk-neutral 

$1 32. 4M 

188.9  mos. 

$65,070.5M 

$1 99.239.4M 

69% 

Risk-averse 

$130. 4M 

186.8  mos. 

$63,690.7M 

$201, 812. 6M 

73% 

Risk-seeking 

$130. 8M 

189.7  mos. 

$65,258.7M 

$202, 934. 9M 

74% 

The  unit  costs  are  not  statistically  different  across  the  three  perspectives. 
There  is  moderate  evidence  that  the  schedule  in  the  risk-seeking  perspective  differs 
from  that  in  the  risk-averse  perspective  (with  p  =  0.065).  For  RDT&E  costs, 
however,  the  cost  from  the  risk-averse  perspective  turned  out  to  be  statistically 
different  from  that  of  the  others.  For  it  and  the  risk-averse  perspective,  p  =  0.026, 
while  for  it  and  the  risk-seeking  perspective,  p  =  0.012.  The  procurement  costs  are 
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not  statistically  different  among  the  three  perspectives.  There  is  not  a  statistically 
significant  difference  among  the  percentages  of  replications  at  which  full  production 
is  achieved  among  the  three  perspectives.  In  conclusion,  for  the  set  of  parameters 
in  this  particular  scenario,  there  is  little  observable  difference  in  outcomes  among  the 
three  different  risk  perspectives.  This  suggests  that  the  three  perspectives,  for  the 
set  of  parameters  operating  in  this  scenario,  select  path  outcomes  that  generate 
relatively  similar  performance  outcomes. 

More  interesting,  though,  is  the  distribution  of  paths  through  the  program 
lifecycle  produced  by  each  of  the  three  alternative  perspectives.  These  are 
summarized  in  Table  8.  Note  that  a  path  pair  is  shown  here  for  each  entry  in  the 
path  distribution.  The  path  pair  consists  of  the  two  paths  with  the  same  set  of 
program  design  decisions,  but  different  outcomes  relative  to  purchase  reduction 
versus  full  production.  For  instance,  paths  13  and  14  have  the  same  set  of  design 
decisions  (prototype  development,  STOVL-first  design  strategy,  no  concurrency,  and 
program  partner  alignment  sought  immediately  after  contract  win).  However,  path 
13  has  full  production,  while  14  has  reduced  purchase.  We  assume  that  the  risk 
perspective  selection  of  paths  relates  to  the  program  design  decisions,  and  not  to 
the  production  level  outcome,  which  is  dependent  on  the  customer  (i.e. ,  government) 
via  probabilities.  The  percentage  of  full  production  outcomes  for  each  set  of 
program  decisions  made  under  each  risk  perspective  is  shown  in  column  4. 

Table  8.  Paths  Generated  by  Different  Perspectives  for  Scenario  1 


Perspective 

Path  Pair 

Frequency 

Full  Production 

% 

Risk-neutral 

13/14 

65% 

64.6% 

15/16 

33% 

75.8% 

9/10 

2% 

0% 

Risk-averse 

13/14 

54% 

74.1% 

5/6 

30% 

76.7% 

15/16 

14% 

64.3% 

7/8 

2% 

0% 

Risk-seeking 

13/14 

69% 

76.8% 

15/16 

28% 

67.9% 

9/10 

3% 

66.7% 

All  perspectives  prefer  development  of  a  prototype.  The  majority  of  selections 
include  the  STOVL-first  design  strategy  (except  for  the  32%  of  the  risk-averse 
perspective’s  selections  that  consist  of  paths  7/8  and  5/6).  The  vast  majority  of 
selections  include  no  concurrency.  The  exceptions  are  paths  9/10  chosen  by  the 
risk-neutral  and  risk-seeking  perspectives.  The  majority  of  paths  selected  include 
the  alignment  effort  at  the  beginning  of  SDD.  However,  a  significant  fraction  do  not 
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(7/8  and  15/16),  indicating  that  this  effort  tends  to  have  less  effect  on  cost  reduction 
than  the  other  design  decisions  and  is  often  overcome  by  the  randomness  in 
outcomes  from  the  simulation’s  probabilistic  nature. 

The  path  selection  outcome  distribution  generated  from  the  risk-averse 
perspective  seems  quite  different  from  that  of  the  other  two  perspectives.  This  is 
confirmed  using  a  chi-squared  test  of  independence  of  categorical  variables.  The 
risk-averse  perspective  is  different  from  both  other  perspectives,  with  p  =  0.000  in 
each  case.  However,  the  evidence  did  not  support  the  conclusion  that  the  risk- 
neutral  and  risk-seeking  perspectives  had  statistically  different  outcomes. 

The  risk-averse  perspective  selected  the  CTOL-first  design  strategy  for  a 
significant  fraction  of  its  paths,  indicating  that  the  STOVL-first  design  strategy  likely 
has  a  probabilistic  risk  of  higher  cost  that  contributes  to  a  greater  maximum 
expected  unit  cost.  The  prototype  option  and  the  no-concurrency  option  seem  not  to 
have  this  issue  (i.e.,  they  are  net  positives  to  unit  cost  in  terms  of  the  expected 
value,  minimum  expected  value,  and  maximum  expected  value). 

Scenario  2 

In  Scenario  1 ,  the  prototype  alternative  is  quite  attractive,  since  it  pushes 
back  PDR  (increasing  schedule  with  a  relatively  low  burn  rate),  while  pushing  up 
CDR  and  production  (decreasing  schedule  with  a  relatively  high  burn  rate).  In 
Scenario  2,  the  prototype  development  alternative  is  made  less  attractive  in  terms  of 
schedule  for  the  program. 

In  the  case  of  the  prototype  relative  to  the  baseline  without  it,  CDR  and 
production  are  pushed  forward  a  period  of  months  triangularly  distributed  ~TR(-9, 

12,  3).  This  means  that  there  is  a  risk  that  the  prototype  may  actually  delay  CDR 
and  production.  Similarly,  for  the  case  of  the  prototype  and  no  concurrency, 
production  is  pushed  forward  months  distributed  ~TR(-15,  4,  -4)  rather  than  ~TR(-9, 
12,  3).  Thus,  production  is  expected  to  be  pushed  back  relative  to  the  baseline  here. 


Table  9.  Average  Simulation  Results  for  Each  Perspective  in  Scenario  2 


Perspective 

Unit  Cost 

Schedule 

RDT&E 

Cost 

Proc.  Cost 

Full 

Production 

Risk-neutral 

$139. 2M 

195.6  mos. 

$63,827.7M 

$21 1 ,1 17.5M 

68% 

Risk-averse 

$133. 8M 

187.5  mos. 

$60,096.7M 

$218, 806. 7M 

76% 

Risk-seeking 

$144. 1M 

214.6  mos. 

$72,278.9M 

$201 ,483. 9M 

62% 

In  this  scenario,  there  are  significant  differences  among  the  outcome 
measures  from  the  three  different  perspectives.  The  unit  costs  of  the  risk-averse 
and  risk-seeking  perspectives  show  significant  difference  with  p  =  0.001 .  There  is 
moderate  evidence  that  the  unit  cost  of  the  risk-neutral  perspective  differs  from  the 
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other  two,  with  p  values  of  0.081  (from  risk-averse)  and  0.130  (risk-seeking) 
respectively.  The  schedule  outcomes  and  RDT&E  cost  outcomes  differed  among  all 
three  perspectives  with  p  =  0.000  for  each  pair-wise  comparison.  Finally,  the 
procurement  costs  showed  significant  difference  between  the  risk-averse 
perspective  and  the  risk-seeking  perspectives  (p  =  0.005).  There  was  moderate 
evidence  of  differences  between  other  pairs  of  procurement  costs,  with  p  =  0.185  for 
risk-neutral  and  risk-averse  and  p  =  0.130  for  risk-neutral  and  risk-seeking. 

One  key  difference  between  this  scenario  and  the  previous  is  the  difference  in 
the  percentage  of  outcomes  that  achieve  full  production  among  the  perspectives. 
Clearly,  the  risk-seeking  perspective  achieves  a  lower  percentage  of  full  production 
outcomes  than  the  other  two.  The  difference  between  it  and  the  risk-averse 
perspective’s  full  production  outcome  is  significant  at  p  =  0.03.  However,  there  is  not 
sufficient  evidence  to  indicate  a  true  difference  between  the  risk-neutral  outcome 
and  the  other  two.  It  is  the  lower  full  production  outcome  for  the  risk-seeking 
perspective  that  is  causing  a  decrease  in  its  procurement  cost,  due  to  fewer  units 
being  produced  on  average,  as  well  as  an  increased  unit  cost,  due  to  having  to 
amortize  RDT&E  and  procurement  costs  over  fewer  units.  This  is  especially  the 
case  since  RDT&E  costs  for  the  risk-seeking  perspective  is  substantially  larger  than 
for  the  other  two  perspectives. 

Once  again,  the  different  perspectives  yield  differing  distributions  of  paths 
selected.  These  are  shown  below  in  Table  10. 


Tab 


e  10.  Paths  Generated  by  Different  Perspectives  for  Scenario  2 


Perspective 

Path  Pair 

Frequency 

Full  Production 

% 

Risk-neutral 

21/22 

31% 

64.5% 

29/30 

31% 

67.7% 

31/32 

20% 

90.0% 

23/24 

17% 

47.1% 

25/26 

1% 

100.0% 

Risk-averse 

21/22 

60% 

81.7% 

23/24 

40% 

67.5% 

Risk-seeking 

13/14 

58% 

60.3% 

15/16 

30% 

63.3% 

9/10 

12% 

80.0% 

Clearly,  the  risk-seeking  perspective  yields  a  much  different  distribution  of 
outcomes  paths  than  do  the  other  two  perspectives.  Also,  the  risk-neutral 
perspective  seems  to  yield  different  path  results  than  the  risk-averse  perspective. 
From  a  statistical  point  of  view,  however,  the  chi-squared  test  of  independence  is 
valid  only  for  the  comparison  of  the  risk-averse  and  risk-seeking  perspectives,  where 
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it  provides  strong  evidence  of  a  difference  in  outcomes  with  p  =  0.000.  It  is  not  valid 
for  the  other  two  comparisons  due  to  numerical  issues.  It  will  be  assumed,  though, 
that  the  other  two  comparisons,  in  fact,  yield  different  results,  and  that  all  three 
perspectives  have  differing  path  outcome  distributions. 

In  this  scenario,  only  the  risk-seeking  alternative  chooses  to  execute  a 
prototype  development,  and  this  is  done  in  100%  of  cases.  It  also  chooses  the 
STOVL-first  design  strategy  in  100%  of  cases.  In  fact,  its  path  outcome  distribution 
appears  similar  to  the  distribution  of  the  risk-seeking  perspective  in  Scenario  1 
(Table  8).  However,  the  chi-squared  test  of  independence  of  categorical  variables  is 
used  to  show  that,  despite  the  same  set  of  outcome  paths,  the  proportions  of  path 
outcomes  are  different  enough  to  declare  that  the  distributions  are  different  at  p  = 
0.04.  Thus,  in  Scenario  2,  the  conclusion  is  that  high-concurrency  is  somewhat 
more  favored  by  the  risk-seeking  perspective  than  in  Scenario  1 ,  due  to  the  higher 
proportion  of  path  pair  9/10. 

The  risk-averse  perspective  chooses  no  prototype,  CTOL-first  design 
strategy,  and  no  concurrency.  Similar  to  Scenario  1 ,  the  alignment  effort  alternative 
is  preferred  (with  60%  of  outcomes),  but  its  effectiveness  at  cost  reduction  seems  to 
be  marginalized  by  the  effect  of  randomness  in  the  simulation. 

The  risk-neutral  perspective  augments  the  risk-averse  perspective’s 
selections  with  a  variety  of  path  pairs,  all  of  which  include  the  STOVL-first  design 
strategy  and  one  of  which  contains  high  concurrency  (25/26),  although  at  only  one 
occurrence. 

Overall,  several  conclusions  can  be  drawn  from  Scenario  2  for  the  set  of 
parameters  studied: 

•  Use  of  a  prototype  is  a  risk-upside  strategy  associated  with  risk¬ 
seeking,  as  is  STOVL-first  design  strategy. 

•  Similar  to  Scenario  1 ,  the  alignment  effort  produces  too  little  cost 
savings  in  RDT&E  to  offset  much  of  the  simulation  randomness. 

•  Risk-seeking  with  this  set  of  parameters  resulted  in  higher  rates  of 
purchase  reduction,  which  drives  lower  procurement  costs,  but  also 
higher  unit  costs.  So  risk-seeking  is  not  an  attractive  strategy  in  this 
scenario. 

Discussion 

The  two  scenarios  analyzed  here  present  different  risks  in  the  program  design 
decisions  of  interest  for  the  F-35  program.  These  risks  are  studied  from  the 
perspectives  of  a  risk-neutral  decision-maker,  a  risk-averse  decision-maker  and  a 
risk-seeking  decision-maker.  In  Scenario  1 ,  most  of  the  risk  of  changing  program 
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design  decisions  from  those  made  in  the  actual  program  is  to  the  upside.  That  is, 
they  tend  to  improve  the  unit  cost  outcome  over  the  span  of  probabilistic  outcomes. 
The  three  perspectives,  therefore,  have  very  similar  performance  outcomes.  In 
addition,  their  path  selection  outcomes  are  not  drastically  different.  Thus,  it  can  be 
concluded  for  the  most  part  that  changing  design  decisions  to  prototype 
development,  STOVL-first  design,  and  no  concurrency  yields  superior  results. 

On  the  other  hand,  Scenario  2  presents  a  more  balanced  set  of  risks  if  design 
decisions  are  changed.  The  different  risk  perspectives  differ  substantially  in  their 
performance  outcomes  and  path  selections.  This  demonstrates  the  capability  of  the 
enterprise  simulation  methodology  to  analyze  a  program  or  enterprise  based  on 
different  risk  perspectives,  so  that  the  decision-maker  can  select  the  one  that  best 
matches  his  or  her  perspective,  while  understanding  outcomes  based  on  other 
perspectives. 

It  should  be  noted  that  the  alignment  effort  tends  to  have  a  positive  effect  on 
unit  costs  in  both  scenarios.  However,  it  is  often  masked  by  the  probabilistic  nature 
of  simulation  outcomes,  since  its  effect  is  not  large.  However,  a  failure  or  significant 
delay  in  attaining  global  design  capability  could  have  political  or  organizational  costs 
that  are  not  modeled  here.  This  is  an  avenue  of  further  research. 

Finally,  the  parameters  used  in  both  scenarios  reflect  estimates  of  what  the 
actual  effects  would  be  from  changing  design  decisions.  A  rigorous  use  of  sensitivity 
analysis  should  be  conducted  in  the  future  to  generate  a  set  of  response  surfaces  to 
characterize  the  conditions  under  which  certain  design  decisions  are  superior  to 
others  under  different  risk  perspectives. 

Conclusions  and  Future  Research 

This  report  has  presented  an  enterprise  simulation  framework  and 
methodology  for  studying  risks  in  defense  acquisition  programs.  The  main  focus  is 
the  use  of  a  decision/event  network  as  an  organizational  story  to  characterize  risk 
drivers  and  outcomes,  as  well  as  to  support  simulation  analysis  of  the  risks  and  risk- 
related  outcomes.  This  is  a  novel  use  of  simulation,  which  tends  to  focus  mostly  on 
experimental  analysis  or  what-if  analysis  for  such  systems  as  acquisition. 

In  particular,  the  enterprise  simulation  methodology  allows  studying  risks  from 
different  risk  perspectives.  This  methodology  is  applied  to  a  case  study  involving 
acquisition  of  the  F-35  Joint  Strike  Fighter.  The  F-35  program  is  the  largest  military 
acquisition  program  in  history,  and  it  has  generated  risks  and  adverse  outcomes,  in 
part  due  to  its  scale,  complexity,  and  transformative  approach  to  acquisition. 
Simulation  analysis  is  used  to  study  the  preferred  outcomes  of  three  different  risk 
perspectives  on  two  different  scenarios.  It  is  shown  that  the  risk  perspectives  can  be 
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used  to  identify  when  a  particular  scenario  has  a  mostly  upside  set  of  risks,  in  which 
case  the  perspectives  tend  to  agree  on  preferred  organizational  stories  and 
performance  outcomes,  and  also  when  a  scenario  has  a  mixture  of  upside  and 
downside  risks.  Similarly,  it  can  be  inferred  that  a  scenario  with  mostly  downside 
risks  can  be  identified  in  this  manner.  Such  analysis  is  especially  important  in 
transformation  efforts,  where  many  risks  are  present  and  may  interact. 

Two  additional  benefits  of  this  methodology  are  the  following: 

•  For  a  complex  scenario  with  many  different  possible  decisions  and 
events,  this  methodology  can  be  used  to  identify  promising  paths 
through  the  decision/event  network,  thus  avoiding  analysis  over  the 
whole  state  space. 

•  A  decision-maker  can  view  the  scenario  through  his  or  her  preferred 
risk  perspective  to  determine  which  outcomes  are  preferred. 

There  are  a  number  of  future  research  avenues  that  can  be  pursued,  both  for 
the  enterprise  simulation  methodology  and  its  application  to  the  F-35  program.  In 
terms  of  the  methodology,  future  research  includes  the  following: 

•  The  number  of  nodes  and  complexity  of  the  network  should  be  scaled 
up  to  allow  analysis  of  computational  issues.  Computational  methods 
can  then  be  developed  to  ameliorate  the  effect  of  large  state  spaces. 

•  The  current  methodology  uses  only  one  performance  measure  in  the 
evaluation  function.  This  could  easily  be  extended  to  a  weighted 
function  of  two  or  more  performance  measures.  It  would  also  be  of 
interest  to  use  other  multi-criteria  decision  approaches  such  as  Pareto 
optimality  or  minimization  of  deviance  from  multiple  performance  goals. 

•  The  methodology  could  be  made  interactive,  with  a  person  as  the 
decision-maker  relying  on  their  interpretation  of  evaluation  functions. 

In  terms  of  the  F-35  program  analysis,  future  research  includes  the  following: 

Additional  socio  factors  could  be  included  beyond  alignment,  such  as 
the  political  and  organizational  cost  of  certain  decisions. 

The  procurement  cost  data  from  the  SAR  is  assumed  to  be  accurate 
for  the  program’s  out-years.  Analysis  could  be  done  to  determine  the 
effect  of  increases  or  decreases  in  these  projected  figures. 

Different  probabilities  for  program  purchase  reduction  could  be 
analyzed  to  determine  their  effect  on  the  different  risk  perspectives. 
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•  Likewise,  a  more  complex  set  of  adverse  outcomes  that  just  a  50% 
purchase  reduction  could  be  studied  (e.g.,  a  range  of  percentages  for 
the  reduction  or  even  a  cancellation  alternative). 

•  The  analysis  could  be  extended  to  sustainment,  which  is  expected  to 
consume  major  funding  once  the  fleet  starts  to  come  online. 
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